Segmentation of an image based on color and color differences

ABSTRACT

An input image is partitioned into a plurality of image regions based on color and color differences. The partitioning comprises assigning a color difference value to plurality of locations within the input image. The partitioning further comprises assigning each of the plurality of locations to an image region of the plurality of image regions, where the assigning occurs according to a particular order. The particular order is based at least in part on color difference values associated with the plurality of locations. The input image may comprise markup. Data representing at least a particular portion of the markup in the input image based on the partitioning is identified. Data representing at least the portion of the markup may be used in a visualization of a customizable product or a manufacturing control associated with a customizable product.

BENEFIT CLAIM

This application claims the benefit under 35 U.S.C. §120 as acontinuation of application Ser. No. 14/203,961, filed Mar. 11, 2014,which claims the benefit of application Ser. No. 14/050,281, filed Oct.9, 2013, which claims the benefit under 35 U.S.C. 119 of application61/785,533, filed Mar. 14, 2013, and application 61/787,532, filed Mar.15, 2013, the entire contents of which are hereby incorporated byreference herein for all purposes as if fully set forth herein. Theapplicant(s) hereby rescind any disclaimer of claim scope in the parentapplication(s) or the prosecution history thereof and advise the USPTOthat the claims in this application may be broader than any claim in theparent application(s).

TECHNICAL FIELD

The present disclosure relates to techniques for segmenting a productmarkup image into regions based on colors and differences betweencolors.

BACKGROUND

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

Information about a digital image, such as the location and shape ofparticular objects in the digital image, may be extracted by performingsegmentation upon the digital image. Image segmentation is a process forpartitioning the digital image into a plurality of different regions.For example, a manufacturer or provider of a custom product, such asZazzle, Inc., may wish to extract the location and shape of markupimprinted on a product for purposes of understanding the geometry of thecustom product, such as a clothing item or accessory, when worn.

One approach for image segmentation is to transform the full-colorrepresentation of an image into a monochrome luminance image where theshade of each of pixel represents the luminance value of the pixel inthe original image. Region partitions may be determined based on theluminance of pixels within the image. However, such an approach oftenproduces inaccurate results since some of the color informationnecessary to determine accurate region partitions is lost in thetransformation to the monochrome luminance image. Approaches forimproved recognition of image partitions in images of marked up productsare needed.

SUMMARY OF THE INVENTION

The appended claims may serve as a summary of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 illustrates an example input image that may be partitioned into aplurality of image regions.

FIG. 2 illustrates an example customization image rendering system whichmay utilize region information determined according to the partitioningprocess.

FIG. 3 illustrates an example process that may be implemented on acustomization image rendering system, such as the customization imagerendering system illustrated in FIG. 2.

FIG. 4 illustrates an example process for partitioning an image intoregions and utilizing the region information that may be determined as aresult of the partitioning process.

FIG. 5 illustrates an example process for partitioning an input imageinto a plurality of image regions based in part on color and colordifferences.

FIG. 6 illustrates an example diagram of location whose colors may beanalyzed to determine a color difference value for a center location.

FIG. 7 illustrates example color difference data, which may be used tosort locations based on their associated color difference values and anexample sort list and color difference array that may be used which mayresult from the sorting process.

FIG. 8 illustrates an example process for sorting image locations basedon color difference values.

FIG. 9 illustrates the contents of a partially-populated sort list andlinked region array which may be populated based on the color differencevalues stored in a color difference array according to the processillustrated by FIG. 7.

FIG. 10 illustrates an example process for assigning image locations toregions. The process may be performed by image processing system 204.

FIG. 11 illustrates example region information that may be determined asa result of the partitioning process.

FIG. 12 depicts a markup grid image that may be determined based in parton the region information identified by partitioning the input imagedepicted in FIG. 1.

FIG. 13 is a block diagram that illustrates a computer system with whichthe techniques herein may be implemented.

DESCRIPTION OF EXAMPLE EMBODIMENTS

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

Embodiments are described herein according to the following outline:

-   -   1. General Overview    -   2. Structural and Functional Overview        -   2.1 Example Customization Image Rendering System        -   2.2 Example Region Information Utilization Processes        -   2.3 Example Markup Image Partitioning Process        -   2.4 Alternatives and Extensions    -   3. Implementation Mechanisms—Hardware Overview    -   4. Example Partitioning Instructions

1. General Overview

Techniques for segmentation of a product markup image based on color andcolor differences are described. Readers of the present disclosure arepresumed to have knowledge and understanding of U.S. application Ser.No. 13/736,844. In an embodiment, the product markup image is an imageof a customizable product comprising markup. The product image may bepartitioned into different regions based in part on color differences.In particular, each location of the product markup image input image maybe assigned a color difference value indicating how different thelocation is from neighboring locations in terms of color. Each locationmay be assigned an image region of the plurality of image regions andlocations may be considered for region assignments according to an orderthat is based, at least in part, on the color difference value assignedto the pixel. For example, locations with low associated colordifference values may be assigned regions before locations with highassociated color difference values.

An image processing system may utilize the determined region informationin visualization of the customizable product or a manufacturing controlassociated with the customizable product. For example, the imageprocessing system may determine the position of particular markupportions based on the region information. Based on the determinedposition information, the image processing system may determine acustomization image rendering asset, which comprises instructions forrendering computer-generated visualization of customized products. Theregion information may also be utilized to verify the quality of acustomized product after the customized product has been manufactured.

2. Structural and Functional Overview

2.1 Example Region Information Utilization Systems

FIG. 1 illustrates an example input image that may be partitioned into aplurality of image regions. Shirt 104 is a customizable product uponwhich custom text or image may be imprinted. Markup 102 is a two-colorcheckerboard pattern comprised of solid-colored squares, where eachsquare is of a single color and each adjacent square is of the oppositecolor. The processes described herein may partition at least the markupportion of the input image into a set of regions, where each region ofthe set of regions represents a single corresponding square of themarkup. In other embodiments, the markup may comprise differentpattern(s), shape(s), or color(s).

The image partitioning process may result in regions information thatidentifies, for example, for each found region, the image locations thatbelong to the region, the area of each region, and/or the representativecolor of the region. In some embodiments, each image location is aseparate pixel. In other embodiments, each image location is a group ofpixels.

In an embodiment, the resulting region information is utilized in theautomatic visualization of a custom product, which may be customizedaccording to customer-provided parameters. FIG. 2 illustrates an examplecustomization image generation and rendering system which may utilizethe region information determined according to the partitioning process.

One or more cameras 202 may capture a photograph of the customizableproduct imprinted or otherwise embellished with the markup. Imageprocessing system 204 may receive and store raw digital images 206received from camera(s) 202. Image processing system 204 comprisescustomized image rendering asset generation instructions 208, which maybe implemented as one or more stored computer programs, scripts, orother software elements. One or more processors of image processingsystem 204 may execute the customized image rendering asset generationinstructions 208 to cause the generation of customization imagerendering instructions. Customized image rendering asset generationinstructions 208 may comprise markup region identification instructions210, which include instructions for analyzing input images to identifypartitioned image regions and collect information about the imageregions, such as a representative color of the region, the area of theregion, and the set of image locations included in the region. Thegeneration of customize image rendering asset instructions may comprisedetermining instructions for generating a customization image renderingbased on the region information. The customization image rendering assetmay contain instructions for rendering a computer-generatedcustomization image that visualizes a customized product.

Image processing system 204 may send a generated customized imagerendering asset to web server 212, such as customization image renderasset 214 to be stored and used at web server 212. Web server 212 mayreceive and store one or more customization image rendering assets, suchas the customization image rendering asset 214. Web server 212 alsocomprises web page generator 216, which determines HTML and/or other webpage content. Web server 212 may receive requests for web pages from auser system, such as user system 220. In response, web server 212 maysend a web page for display at the user system, and may include acustomization image that is generated by the execution of instructionsincluded within the customization image rendering asset. Web server 212may be operated by an on-line retailer for the generation of onlineretail web sites.

User system 220 may be the system by which a customer browses thewebsite of an online retailer, selects a product to customize, specifieshow the product is to be customized, and receives the customizationimage. User system 220 may request web pages and web content form webserver 212 via network 28, which may be the World Wide Web (WWW).

The system illustrated in FIG. 2 is merely an example of customizationimage rendering system. Systems of other embodiments may not includeeach of the components and sub-components illustrated in FIG. 2.Further, for the purpose of clearly illustrating an example, FIG. 2shows individual instances of various functional elements as describedabove; however, in various embodiments, each functional element may beimplemented as one or more local or distributed computers, processes,software elements or other logic.

2.2 Example Region Information Utilization Process

FIG. 3 illustrates an example process that may be implemented on acustomization image rendering system, such as the customization imagerendering system illustrated in FIG. 2. According to the process of FIG.3, a customization image rendering asset, which contains instructionsfor the generation of a customization image, is determined based in parton region information determined according to the partitioning process.The process of FIG. 3 may be performed by image processing system 204.

At block 302, photograph(s) of a product are captured. The photographedproduct maybe visibly imprinted or otherwise embellished with markup. Insome embodiments, the product being photographed is modeled by a personor animal. The photograph(s) may be captured at one or more cameras suchas camera(s) 102. At block 304, a photograph collection and processingsystem, such as image processing system 204 of FIG. 2, receives thecaptured photograph(s) of the product. At block 306, customization imagerendering asset generation instructions are executed at the photographcollection and processing system. The execution of the customizationimage rendering asset generation instructions cause the generation of acustomization image rendering asset.

At block 308, the customization image rendering asset is sent to the webserver. Multiple products may be photographed individually, and adifferent customization image rendering asset may be created for each ofthe multiple products at the photograph collection and processingsystem, and each of the customization image rendering assets may be sentto the web server.

At block 310, the customization image rendering assets are received andstored at a web server, such as web server 112. The web server may storevarious customization image rendering assets. At block 312, the webserver receives a customer customization specification. The customer mayhave provided the specification by interacting with a graphicalinterface presented within a web page viewed at a user system. The usercustomization specification may include a user-specified design forimprinting on the customizable product, customization locationinformation specifying where a particular customization is to belocated, a color for a detail area of the product such as a trim color,a color for the entire product, and other customization specificationinformation.

At block 314, the instructions of a particular customization imagerendering asset are executed. The web server may store a customizationimage rendering asset for each customizable product. The web server mayautomatically select a particular customization image rendering assetfor execution of its instructions based on the received customizationspecification. For example, the web server may select a particularcustomization image rendering asset that corresponds to a particularcustomizable product in response to determining that the customerselected to customize the particular customizable product. Informationprovided by the user, and included in the received user customizationspecification may be used to automatically determine inputs for thecustomization image rendering asset. For example, a customer may selecta particular shirt color and the selected color, or a color curvecorresponding to the selected color, may be provided as an input to thecustomization image rendering asset instructions when the instructionsof the customization image rendering asset are executed. Executing theinstructions of the customization image rendering asset results in acustomization image that depicts the customer-selected product ascustomized according to the customer's specifications.

At block 316, the web server sends web page content to the user systemthat provided the customization specifications, which may be user system120. The web page content includes a customization image determined as aresult of executing the instructions of the customization imagerendering asset, or a modified version of such an image. At block 320, aweb page that includes the received customization image is displayed atthe user system.

FIG. 3 illustrates merely one example process that may be implemented ona customization image rendering system. In other embodiments, differentsteps may be performed, the ordering of steps may be changed, certainsteps may not occur, or additional steps may occur in addition to thesteps illustrated in FIG. 3.

FIG. 4 illustrates an example process for utilizing the regioninformation that may be determined as a result of the partitioningprocess in the visualization of a customizable product. The process ofFIG. 4 may be performed by image processing system 204. At block 402, aninput image of a customizable product comprising markup is received.Image processing system 204 may modify the input image before beginningthe partitioning process for improved segmentation. For example, imageprocessing system 204 may remove luminance information from the inputimage before the partitioning.

At block 404, the input image is partitioned into a plurality of imageregions based in part on color and color differences. FIG. 5 illustratesan example process for partitioning an input image into a plurality ofimage regions based in part on color and color differences.

FIG. 11 illustrates example region information that may be determined asa result of the partitioning process. Region list 1108 includes regionrecords for each determined region. For each region, region list 1108identifies an image location that has been assigned to the region, i.e.location ID 1110, a representative color for the image region, which maybe an average color of the region or the color of the first locationassigned to the region, i.e. region color value 1112, and an area of theregion, i.e. region area value 1114. According to different embodiments,the region records may include additional or different types ofinformation such as the number of locations included in the region whichare edge locations that are located on the edge of a region. The IDcorresponding to a particular region may be determined based on thelocation of the entry corresponding to the particular region in theregion list. For example, the first entry of region list 1108 maycorrespond to the region identified by a region ID of “1.”

Region list 1108 may be any of a plurality of data structures including,but not limited to, a linked list, array, or an array of linked lists.

Linked region array 1102 illustrates an example final form of the linkedregion array according to one embodiment. For each linked region arrayentry of linked region array 1102, the region ID field may identify theregion assigned to the location corresponding to the linked region arrayentry and the next location field may identify a next location of theinput image which has been assigned to the same region as the locationcorresponding to the linked region array entry. For example, region IDfield 1104 corresponds to the second location of the input image and thevalue contained in region ID field 1104, “7,” indicates that the secondlocation of the input image has been assigned to region 7. Next locationfield 1106 also corresponds to the second location of the input imageand the value contained in next location field 1106, “3,” indicates thatthe third location of the input image has also been assigned to the sameregion as the second location of the input image.

At block 406, data representing at least a particular portion of themarkup in the input image is identified based on the partitioning. Thedata representing the portions of the markup may be the set of imagelocations that are within the particular portion of the markup. The datamay be determined based on the determined region information, such aslinked region array 1002 and region list 1008.

Image processing system 204 may identify the particular image regionsthat correspond to the particular portion based on region recordinformation contained in region list 1008. For example, if the markup isa checkerboard pattern of yellow and purple, image processing system 204may efficiently locate all yellow portions of the markup by identifyingall regions whose region color is similar to yellow, and thenidentifying the locations which have been assigned to those regions.Image processing system 204 may identify all regions whose region colorvalue is similar to yellow by calculating a color difference value foreach region represented in a region list, where the color differencevalue for the region indicates how different the region's representativecolor, as indicated in region list 1008, is from the color yellow. Eachof the regions whose corresponding color difference value is less than athreshold amount may be identified as a yellow region and each locationbelonging to that region may be determined to be yellow.

The entire set of image locations belonging to a particular region maybe efficiently determined by looking up the location ID of the firstlocation in the corresponding location ID field of the region list 1008and the remaining locations may be determined by traversing the nextlocation fields of linked region array 1002, beginning with the nextlocation field corresponding to the location identified by the locationID contained region list 1008. For example, each of the locationsbelonging to region 7 may be determined by looking up location ID 1010in region list 1008, whose value is “2.” The next location belonging toregion 7 may be determined by looking up the next location fieldcorresponding to the second image location, next location field 1006,which contains the value “3.” The next location belonging to region 7may be determined by looking up the next location field corresponding tothe third image location, next location field 1006, which contains thevalue “0.” In an embodiment a value such as “0” may be used to indicatethat there are no further image locations belonging to the region.

The data representing at least the particular portion may indicate otherinformation, such as the color of the particular portion of the markuppattern in the input image. For example, once image processing system204 determines the location(s) of the input image that belong to theparticular portion of the markup, image processing system 204 may accessthe original input image to determine color information for thedetermined locations.

At block 408, use of the data representing at least the portion of themarkup in a visualization of the customizable product or a manufacturingcontrol associated with the customizable product is caused. Causing useof the data representing at least the portion of the markup in avisualization of the customizable product may comprise determining acustomization image rendering asset comprising instructions for therendering of a customization image based on the data, where thecustomization image rendering asset may be sent to a web server forsubsequent use by the web server. Causing use of the data representingat least the portion of the markup in a visualization of thecustomizable product may also comprise determining a customization imagebased on the data.

In an embodiment, customization image rendering asset 214 renders acustomization images that visualize a customized product as it wouldappear in when in use, with natural contouring and shape. For example,the coloring and/or shape of the customized product in the customizationimage may depict the folds or natural contours that occur when thecustomized product is worn by a model. Image processing system 204 maydetermine a customization image rendering asset which generates suchcustomization images based on an analysis of an input image such as theinput image depicted in FIG. 1.

Image processing system 204 may utilize the region informationdetermined by the partitioning process to determine how the particularportion of the input image differ with regards to color or location fromthe same particular portions of a sample reference image depicting thesame markup. For example, the lines of each square of markup 102 mayappear straight and parallel to each other in the sample referenceimage, whereas they appear slanted and disjoined in some areas of theinput image, such as at areas where the customizable product is twistedor folded. The sample reference image may depict the same markupdepicted in the input image, except as a flat image. Image processingsystem 204 may map locations in the sample reference image to locationsin the input image and compare how the position and color of the twolocations differ in the two images. Image processing system 204 maygenerate customization image rendering asset 214 based on an analysis ofhow the input image and sample reference image differ.

Customization image rendering asset 214 may render the customizationimage based on a flat customization-specification image depicting thecustomization to be applied to a customizable product. Thecustomization-specification may be a customer-uploaded image, design, ortext that is to be imprinted on the customizable product. Customizationimage rendering asset 214 may generate the customization image in partby modifying the customization-specification image to depict the foldsand contours as depicted in the input image.

In an embodiment, image processing system 204 determines a markup gridimage based on the region information, where the markup grid imageidentifies the edges between different markup portions. For example,image processing system 204 may determine a markup grid image based inpart on the region information identified by partitioning an inputimage. FIG. 12 depicts a markup grid image that may be determined basedin part on the region information identified by partitioning the inputimage depicted in FIG. 1. Image processing system 204 may determinecustomization image rendering asset 214 based in part on the markup gridimage.

Example processes for utilizing a markup grid image to determine acustomization image rendering asset may be further described in U.S.application Ser. Nos. 13/736,844, 13/342,103, 12/790,711 and U.S. Pat.Nos. 8,174,521, 8,175,931, 8,090,461, although other processes may beused in different embodiments. The contents of all patent applicationsand patents cited in this paragraph are hereby incorporated by referencein their entirety for all purposes as if fully set forth herein.

The lines of the markup grid may correspond to the shape of the bordersbetween the squares in the markup of the color segmented image. Theshapes of the lines in the markup grid indicate the geometry of theproduct in the design areas. For example, a slanted line may indicatethat the product is slanted at the corresponding location. Suchinformation may indicate how a customization design would appear whenprinted in the design region of a customized product. In an embodimentwhere there may be multiple design areas, design geometry informationmay be determined for each of the design areas. Although the markup gridimage of FIG. 12 is illustrated as roughly parallel lines, in otherembodiments, the markup grid image may be any of a variety of differentshapes and/or patterns determined based on the markup. In an embodiment,a grid determined based upon a flat design image is mapped to a griddetermined based upon the markup grid image to determine instructions tocompare how various locations of the images differ in position and/orcolor in the two image.

The markup grid image of FIG. 12 may be analyzed to determine acustomization image rendering asset. The customization image renderingasset may comprise instructions for rendering an image depicting aproduct customized according to user specifications. The user mayprovide a user image to be imprinted upon a product and thecustomization image rendering asset may depict how the product wouldappear with the image imprinted upon the product. The customizationimage generated by the customization image rendering asset may depictthe same geometry and contours of the example input image. For example,the customization image may depict the shirt as being folded in the samelocation as depicted in the example input image.

FIG. 4 illustrates merely one example process of identifying andutilizing region information. In other embodiments, different steps maybe performed, the ordering of steps may be changed, certain steps maynot occur, or additional steps may occur in addition to the stepsillustrated in FIG. 4.

In other embodiments, the determined region information may be utilizedin the manufacturing control of a customized product. “Manufacturing” asused herein may refer to the imprinting of a particular customization ona pre-manufactured product. For example, the region information may beutilized to determine the location of a particular customization portionin the customized product and to ensure that that the color and positionof the particular customization portion is within an expected range ofcolors and positions.

As another example, a provider of a custom product may providecustomer-specified specifications for a custom product to a manufacturerwho manufactures the customized product. When introducing a new productfor custom imprinting, the manufacturer may specify an imprinting layoutof their product, where the imprinting layout specifies the locations ofthe product upon which designs may be imprinted. For example, themanufacturer of a mobile phone case may specify that a particular regionof the mobile phone case corresponding to a camera hole in the mobilephone case is a region at which a custom design may not be imprinted.The imprinting layout may specify that the region below the camera holemay be imprinted with a custom design.

In response to receiving the imprinting layout of the product from themanufacturer, the provider may send a mark-up image to the manufacturerfor imprinting on the custom product. The manufacturer may imprint themark-up image upon the product and send the customized product to theprovider. The provider may visually examine the custom imprinted productto ensure that the imprinting of the custom design relative to thefeatures of the product is visually appealing. The provider may makeadjustments to the imprinting layout based on the examination. Forexample, a manufacturer may have failed to specify that a custom designmay not be imprinted on a location of the mobile phone casecorresponding to the camera hole. Upon receiving the custom imprintedproduct, a provider may determine that the placement of the design uponthe mobile phone case is not visually appealing since a portion of thedesign is missing due to its location overlapping the camera hole. Theprovider may modify the size the design region to not cover the camerahole region or may shift the design region to a different location uponthe phone. In an embodiment, the provider may propose an alternateimprinting layout to the manufacturer for review and approval by themanufacturer.

2.3 Example Markup Image Partitioning Process

FIG. 5 illustrates an example image partitioning process. The process ofFIG. 5 may be performed by image processing system 204. At block 502, acolor difference value is assigned to each location of a plurality oflocations within the input image. In some embodiments, each location isa separate pixel of the input image and each pixel of the input image isassigned a color differee value. A color difference value indicates howdifferent the color of the corresponding location is from colors ofneighboring locations.

In an embodiment, the color difference value for a particular locationis determined based on a comparison of the color of the particularlocation and the color of locations that are distances of 2, 4, and 6locations away. A color difference may value may be calculated for eachdistance and the three color values, each associated with a differentdistance, may be averaged to determine the final color difference value.Averaging the color difference values may comprise weighting colordifference values associated with the greater distance more heavily thancolor difference values associated with the lesser distance. A colordifference value may be calculated for a distance by determining howdifferent an average of the colors of the different sampled locations ofthat distance is from the color of the center pixel. The colordifference value for two locations may be calculated by calculating adot product of the difference between the two colors.

FIG. 6 illustrates an example diagram of location whose colors may beanalyzed to determine a color difference value for a center location. Inan embodiment, a color difference value for center location 602 iscalculated using Equation 1:

$\begin{matrix}{{{ColorDiffVal}_{Center} = {{\frac{1}{6}{{ColorDiff}( {{Color}_{{{Dia}{\lbrack 2\rbrack}}{\_ {Average}}},{Color}_{Center}} )}} + {\frac{1}{3}{{ColorDiff}( {{Color}_{{{Dia}{\lbrack 4\rbrack}}{\_ {Average}}},{Color}_{Center}} )}} + {\frac{1}{2}{{ColorDiff}( {{Color}_{{{Dia}{\lbrack 6\rbrack}}{\_ {Average}}},{Color}_{Center}} )}}}}{{Color}_{{{Dia}{\lbrack 2\rbrack}}{\_ {Average}}} = ( \frac{{Color}_{{Dia}\mspace{14mu} {2{\lbrack 1\rbrack}}} + \ldots + {Color}_{{Dia}\mspace{14mu} {2{\lbrack 7\rbrack}}}}{7} )}{{Color}_{{{Dia}{\lbrack 4\rbrack}}{\_ {Average}}} = ( \frac{{Color}_{{Dia}\mspace{14mu} {4{\lbrack 1\rbrack}}} + \ldots + {Color}_{{Dia42}{\lbrack 11\rbrack}}}{7} )}{{Color}_{{{Dia}{\lbrack 6\rbrack}}{\_ {Average}}} = ( \frac{{Color}_{{Dia}\mspace{14mu} {6{\lbrack 1\rbrack}}} + \ldots + {Color}_{{Dia}\mspace{14mu} {6{\lbrack 7\rbrack}}}}{7} )}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

In other embodiments, determining color difference values may comprisesampling more or less locations, sampling locations at differentdistances, and/or assigning different weights to the determined colordifference values.

At block 504, the color difference values are smoothed to decrease thevariation between color difference values. In an embodiment, onlyrelatively high and relatively low color difference values are modifiedto be, respectively, lower and higher. For example, only colordifference values that are in the top 10 percent of highest colordifference values or in the lowest 10 percent of color difference valuesmay be modified. In an embodiment, a monochrome image is created basedon the input image, where the color of each pixel represents the colordifference value of the pixel. Smoothing the color difference values maycomprise applying a Gaussian filter to the monochrome color differencevalue image. Smoothing color difference values before region assignmentmay prevent over-segmentation of the input image.

At block 506, locations are sorted based on their associated colordifference values. The sorting may comprise grouping image locationsinto different groups based on their associated color difference values.FIG. 8 illustrates an example process for sorting image locations basedon color difference values. At block 508, locations are assigned toregions, and the assignment of locations to regions occurs based, atleast in part, on sort order. FIG. 10 illustrates an example process forassigning image locations to regions.

Image processing system 204 may analyze and partition a portion of aninput image into regions or may analyze and partition an entire inputimage into a plurality of image regions. In an embodiment where imageprocessing system 204 analyzes and partitions a portion of an inputimage into regions embodiment, color difference values and regionassignments may only be determined for the locations within the portionof the input image region.

FIG. 5 illustrates merely one example image partitioning process. Inother embodiments, different steps may be performed, the ordering ofsteps may be changed, certain steps may not occur, or additional stepsmay occur in addition to the steps illustrated in FIG. 5.

FIG. 7 illustrates example color difference data, which may be used tosort locations based on their associated color difference values and anexample sort list and color difference array that may be used which mayresult from the sorting process. Sort list 712 and linked region array706 may be populated based on color difference array 702.

Color difference array 702 identifies the color difference valuesdetermined for each location in the input image. Each entry of colordifference array 702 corresponds to a particular location of the inputimage and the value in the entry identifies the color difference valuefor the corresponding location of the input image.

Sort list 712 may comprise an entry for each color difference valuemagnitude included in color difference array 702. For example, if thecolor difference values in color difference array 702 range from 1-5,sort list 712 may comprise five entries. According to an embodiment,after sort list 712 is fully populated, each entry of sort list 712 isassociated with a particular magnitude and includes a value that either(1) identifies an image location that is associated with the particularcolor difference value and/or linked region array entry whichcorresponds to an image location that is associated with the particularcolor difference value or (2) indicates that no location of the inputimage has a color difference value of the particular magnitude (i.e. avalue of “0”). For example, sort list entry 714, which is the firstentry of sort list 712, may be associated with the color differencevalue magnitude “1.” Sort list entry 714 contains the value “12,” whichidentifies the twelfth location of the linked region array 706, linkedregion array entry 712, as well as twelfth location of the input image.The twelfth location of the input image is associated with colordifference value 704 of “1,” as indicated by the color difference value704 value of “1.”

Linked region array 706 may contain an entry for each location of theinput image, where each entry corresponds to a particular location. Eachentry may include a region ID field and a next location field. After theregion ID fields of linked region array 706 are fully populated, eachregion ID field may contain a value identifying the region to thecorresponding locations is assigned. For example, region ID field 708correspond to the second location of the input image and the value “7”contained in region ID field 708 indicates that the second location ofthe input image is assigned to region 7. The region ID fields of linkedregion array 706 may remain empty, or may be initialized to 0 or anothervalue, until the region assignment process occurs.

After the next location fields of linked region array 706 are fullypopulated according to the sorting process, the value in each particularnext location field either (1) identifies another image location that isassociated with same color difference value as the input image locationassociated with the particular “next location” and/or another linkedregion array entry which corresponds to another location that isassociated with same color difference value as the input image locationassociated with the particular next location field or (2) indicates thatthere are no further linked region array entries that correspond toimage locations having the same color difference value as the inputimage location to which the particular next location field corresponds.

In an embodiment, image processing system 204 maintains mappings whichidentify, for each image location, the linked region array entry andcolor difference array entry that corresponds to the input imagelocation. In another embodiment, the ordering of data in the linkedregion array 706 and color difference array 702 correspond to theordering of image locations, such that a single location ID identifies aparticular color difference array entry in color difference array 702, aparticular linked region array entry in linked region array 706, and aparticular image location. For example, the value “12” in sort listentry 714 identifies both the twelfth entry of color difference array702 and the twelfth entry of linked region array 706, which bothcorrespond to the twelfth location in the input image. The value “12” insort list entry 714 also identifies the twelfth location in the inputimage.

Sort list 712 and linked region array 706 may be used during the regionassignment process to efficiently locate all image locations that areassociated with a particular color difference values and/or theircorresponding linked region array entries. The various linked regionarray entries that correspond to image locations with the same colordifference values may be linked via next location fields of the linkedregion array. Each location associated with a particular colordifference value may be efficiently identified by first looking up thesort list entry corresponding to the particular color difference valuein sort list 712, which identifies the am image location correspondingto the color difference value. The remaining image locations associatedwith the particular color difference value may be identified byfollowing the next location fields of linked region array 706, each ofwhich identify the next image location associated with the particularcolor difference value or the next linked region array locationcorresponding to the next image location associated with the particularcolor difference value.

For example, each input image location corresponding to color differencevalue “1” may be identified by accessing the sort list entrycorresponding to the value “1,” which is the first entry in sort list712, sort list entry 714. Sort list entry 714 identifies the first imagelocation corresponding to the color difference value of “1.” Since sortlist entry 714 contains a value of “10,” image processing system 204 maydetermine that the twelfth location of the input image is associatedwith a color difference value of “1.” The next input image locationassociated with a color difference value of “1,” may be determined byaccessing next location field 710, which is the “next location field”corresponding to twelfth location in linked region array 706. Nextlocation field 710 may contain a value, such as “0,” which indicatesthat there are no further locations associated with the color differencevalue of “1.” In another embodiment, the next location fieldcorresponding to the twelfth location may identify another imagelocation associated with the color difference value of “1” or anotherlinked region array entry corresponding to an image location associatedwith the color difference value of “1.”

FIG. 8 illustrates an example process for sorting image locations basedon color difference values. The process comprises populating a sort listand linked region array based on a color difference array. The processmay be performed by image processing system 204. Image processing system204 may iterate through the values of a color difference array, each ofwhich are associated with a corresponding image location, and store alocation identifier identifying the corresponding image location ineither the sort list or linked region array. The sorting may comprisegrouping together image locations with the same color difference values,for example by adding the location identifier of a particular locationto a list of location identifiers identifying other locations of thesame color difference value.

FIG. 9 illustrates the contents of a partially-populated sort list andlinked region array which may be populated based on the color differencevalues stored in a color difference array 702 according to the processillustrated by FIG. 8. Sort lists 902A, 902B and linked region arrays906A, 906B illustrate different states of the same sort list and linkedregion array, which are populated based on color difference array 702and according to the process of FIG. 8. Sort list 902A and linked regionarray 906A illustrate the sort list and linked region array after asingle iteration of the process illustrated in FIG. 7. Sort list 902Band linked region array 906B illustrate the sort list and linked regionarray after two iterations of the process illustrated in FIG. 8. Priorto initiating the process of FIG. 8, sort list 902A and linked regionarray may only contain their initialized values. For example, sort list902A and linked region array 906A may not have contained any valuesprior to initiating the process of FIG. 4 or may have all containedvalues of “0.”

At block 802, image processing system 204 determines the colordifference value for the current location. At the beginning of theprocess, the current location may be the first location in the inputimage. The color difference value for the current location may bedetermined by accessing the value in the color difference array entrycorresponding to the current location. In some embodiments, the colordifference values in the color difference array are ordered accordingthe ordering of the locations to which the color difference valuescorrespond. For example, the color difference value associated with thefirst image location maybe the value in the first entry of the colordifference array.

Next, at block 804, image processing system 204 determines whether thesort list entry corresponding to the determined color difference valuealready contains a location ID. Determining whether the sort list entrycorresponding to the determined color difference value contains alocation ID may comprise determining whether the sort list entrycontains a value different than the initialized value. For example, sortlist 902A may be initialized to contain values of “0” and determiningwhether the sort list entry corresponding to the determined colordifference value contains a location ID may comprise determining whetherthe sort list entry contains a non-zero value.

If image processing system 204 determines that the sort list entrycorresponding to the determined color difference value does not containa location ID, the process proceeds to block 806 and the location ID ofthe current location is stored in the sort list entry corresponding tothe determined color difference value. The sort list may not contain alocation ID if no location with the same color difference value as thecurrent location was previously considered during the sort list andlinked region array population process. For example, during the firstiteration of the process, image processing system 204 may determine thecolor difference value for the first image location by locating thefirst value in color difference array 502, which is “4.” Accordingly,the fourth entry in sort list 902A, sort list entry 904A, may be updatedto store the value “1,” which is the location ID of the first imagelocation. Sort list entry 904A may previously have contained “0” or someother initialization value.

At block 812, image processing system 204 determines if the currentlocation is the last location of the input image. If the currentlocation is not the last location of the input image, the processproceeds to block 814 and the current location is incremented to be thenext location of the input image. For example, after the first iterationof the process is complete, the current location may be incremented tobe the second location of the input image and the process of FIG. 8 mayrepeat with the current location of the input image being the secondlocation of the input image.

At block 802, the color difference value of the new current location maybe determined to be “4” by locating color difference value 716 in colordifference array 702. At block 804, the process determines whether thesort list entry corresponding to the determined color difference valuealready contains a location ID. The sort list entry corresponding to thevalue “4,” sort list entry 904A, contains a location ID of “1.” Thus,the process proceeds to block 808 because the sort list entrycorresponding to the determined color difference value already containsa location ID.

At block 808, the location ID already stored in the sort list entry ismoved to the next location field corresponding to the current locationin the linked region list. For example, the value “1,” which was storedin sort list entry 904A is moved to next location field 908, which isthe next location field corresponding to second image location in linkedregion array 906B. At block 810, the location ID of the current locationis stored in the sort list entry corresponding to the determined colordifference value in the sort list array. For example, the value “2,”which is the location ID of the second location of the input image isstored in the sort list entry corresponding to the color differencevalue of “4,” sort list entry 904B.

At block 812, image processing system 204 determines if the currentlocation is the last location of the input image. If the currentlocation is the last location of the input image, the process ends atblock 816. If the current location is not the last location of the inputimage, the process proceeds to block 814 and the current location isincremented to be the next location of the input image. The processrepeats with each location until there are no further image locations toanalyze. After the process of FIG. 8 is complete, the sort list and nextlocation fields of the linked region array may be fully populated withlocation IDs for each location of the input image.

The linked region array and the sort list may be used during the regionassignment process to traverse information for each location, where thelocations are considered according to an order based on the locations'color difference values. In an embodiment, locations with low colordifference values, i.e. locations that are very similar in color totheir neighboring locations, are assigned regions before locations withhigh color difference values. For example, for an image whose determinedcolor difference values range from 1-5, image processing system 204 mayfirst assign regions to each location whose color difference value is 1,followed by region assignments for each image location whose colordifference value is 2, and so forth until all locations have beenassigned regions.

According to an embodiment, an image location is more likely to beassigned a new region, to which no other image locations have beenassigned, if relatively few image locations have been assigned region.For example, if one or more image locations that neighbor the inputimage location under consideration have already been assigned regions,the region for the input image location under consideration is selectedfrom one of the neighboring regions. If the input image location underconsideration only neighbors image locations that have not been assignedregion, the input image location may be assigned a new image region towhich no image locations have been assigned. According to such aprocess, the input image locations that are considered for regionassignment first may be more likely to be at the center of thedetermined regions or other non-border locations of the regions. Thus,by assigning region to locations based on their color difference values,locations with low color difference values (i.e. locations that are verysimilar to their surrounding locations) may be more likely to be at thecenter of the determined regions or in other non-border locations of theregions. Such an image partitioning approach may cause the resultingpartitioned image to be partitioned into regions that are moreuniformly-colored than regions determined according to alternateapproaches.

FIG. 8 illustrates merely one example process for sorting imagelocations based on color difference values. In other embodiments,different steps may be performed, the ordering of steps may be changed,certain steps may not occur, or additional steps may occur in additionto the steps illustrated in FIG. 8.

FIG. 10 illustrates an example process for assigning image locations toregions. The process may be performed by image processing system 204. Atblock 1002, a location of the input image is selected as currentlocation. According to an embodiment, determining a current locationcomprises accessing either a sort list entry or linked region arrayentry to determine the first or next image location. During the firstiteration of the process of FIG. 10, the first current location may bedetermined by accessing the sort list entry corresponding to the lowestcolor difference value magnitude of all found color difference values.The value contained in the sort list entry may identify an imagelocation that has the lowest color difference value of the input image.For example, a first current location may be determined by accessingsort list entry 514 in sort list 512. Based on the location ID “12”contained in sort list entry 514, image processing system 204 maydetermine that the twelfth location of the input image is the firstcurrent location.

At block 1004, the set of regions that neighbor the current location areidentified. The set of regions that neighbor the current locationinclude all regions to which one or more location that neighboring thecurrent location are assigned. For example, if a location neighbors fourother locations, one of which has been assigned region “1” and two ofwhich have been assigned region “2,” the set of regions that neighborthe current location include “1” and “2.” The particular locations thatare considered neighboring locations may vary according to differentembodiments. For example, neighboring locations may include all the fourlocations directly above, below, to the left, and to the right of thecurrent location. In another embodiment, neighboring locations mayadditionally include the four locations diagonally adjacent to thecurrent locations.

Next, at block 1006, image processing system 204 determines if thenumber of different neighboring regions equal 0. If so, image processingsystem 204 performs the action of block 1008, and assigns the currentlocation to a new region, to which no other locations are assigned. Ifimage processing system 204 determines that the number of differentneighboring regions do not equal zero, at block 1010, image processingsystem 204 determines if the number of different neighboring regionsequal one. If so, at block 1012, image processing system 204 assigns thecurrent location to the single neighboring region.

If image processing system 204 determines that the number of differentneighboring regions do not equal zero or one, image processing system204 determines, at block 1014, if the number of different neighboringregions equal two. If so, at block 1016, image processing system 204determines if the two neighboring regions are similar enough to bemerged. Determining whether two neighboring regions are similar enoughto be merged may comprise comparing a representative color of one regionto the representative color of the other region.

According to one embodiment, two colors are similar enough to be mergedif the dot product of their difference is less than a particularthreshold difference amount. For example, if the representative colorsfor the two regions are (A,B,C) and (X,Y,Z) the two regions may bemerged if Expression 1 is true:

((A−X),(B−Y),(C−Z))·((A−X),(B−Y),(C−Z))<Threshold Amount  Expression 1

In some embodiments, the color of the image is represented in threedimensions. For example, color may be expressed using the RGB modelcomprising a three-dimensional color value. In other embodiments, thecolors of the image may be represented in a different number ofdimensions. For example, color may be expressed using thefour-dimensional CMYK color model. Thus, the processes described hereinneed not be limited to images whose colors are expressed in a particulardimensionality.

According to various embodiments, the representative color may be theaverage color of the location assigned to the regions thus far or thecolor of the first location assigned to the region. The representativecolor for a region may be determined by accessing a region record forthe region that is updated in response to each addition of a newlocation to the region.

If image processing system 204 determines that the two regions aresimilar enough to be merged, at block 1022, image processing system 204merges one region of the two regions into the other region of the tworegions. In an embodiment, the region which was discovered later of thetwo is merged into the region that was discovered earlier of the two.

In an embodiment, assigning a location to a region comprises storing avalue identifying the region in an entry of a region list correspondingto the location. The first location that is assigned a region may beassigned a region with a region ID of “1.” The next time a location isto be assigned to a new region that has not previously been assigned anylocations, the region ID of “2” may be selected as the region ID of thenew region. Thus, each newly assigned region may be associated with ahigher numerical value than the regions that have already been assignedto one or more locations. According to such an approach, the newerregion of two regions may be identified by comparing the region IDs ofthe two regions.

In an embodiment where the region which was discovered later of the twois always into the region that was discovered earlier of the two, aparticular region may be selected as the region that retains its regionID after the merger based on a comparison of the region ID valuesidentifying the two regions being merged. The region having the regionID of the lower numerical value may be selected as the region thatretains its region ID. The other region maybe “merged into” the selectedregion by each instance of the region ID in the region list beingreplaced by the region ID that identifies the selected region.

For example, if two regions are being merged and a first region ID isassociated with a region ID of “10” and the other region is associatedwith a region ID of “14,” the region corresponding to the region ID of“14” is merged into the region corresponding to the region ID of “10.”Each occurrence of the value of “14” in the region list may be replacedby the value of “10.”

In other embodiments, each new region may be assigned a number ofincreasing value and the region with the higher region ID may beselected as the region that the other region is to be merged into.

In one embodiment, in response to the determination to merge the regionof one neighboring location with the region of the other neighboringlocation, the current location is assigned to the region that isselected to be merged into by the other region (“the selected region”).In an embodiment where assigning a location to a region comprisesstoring a value identifying the region in an entry of a region listcorresponding to the location, a value identifying the selected regionis stored in the entry corresponding to the current location. In oneembodiment, every other value identifying the non-selected region in theregion list is immediately replaced by the value identifying theselected region in response to the determination to merge the tworegions. “Immediately” in this context indicates that no other valuesare considered for region assignment until the values are upated.

In another embodiment, only the entry corresponding to the currentlocation is updated immediately in response to the determination. Anindication of which regions were merged into other regions may be storedand the remaining entries in the region list that contain the region IDsidentifying the non-selected region are updated to include the region IDidentifying the selected region after each location in the image hasbeen assigned a region ID.

In an embodiment where the region with the higher value region IDconsistently adopts the region ID of the region with the lower valueregion ID, or where the opposite is consistently true, the updating ofregion IDs for merged regions may be completed in a single traversal ofthe list storing region IDs for each location.

At block 1024, after the two regions are merged, the current location isassigned to the region into which the other region was merged.

If image processing system 204 determines that the two regions are notsimilar enough to be merged, at block 1018, image processing system 204assigns the current location to the region most similar in color to thecurrent location based on comparisons of the current location's colorand the representative colors for the neighboring regions. In anembodiment, two color difference amounts are calculated, one colordifference amount being relative to the color of the current locationand the representative color of one region and the other colordifference amount being relative to the color of the current locationand the representative color of the other region. A color differenceamount relative to two colors may be determined by calculating the dotproduct of the difference between the two colors. The region from whichthe current location is the least different in terms of color may beidentified as the selected region.

Image processing system 204 may proceed to block 1020 if imageprocessing system 204 determines that the number of neighboring regionsis not equal to 0, 1, or 2. In such a case, the current locationneighbors at least three different regions. At block 1020, imageprocessing system 204 assigns the current location to the region of theset of neighboring regions whose representative color is most similar incolor to the current location. Such a region may be identified bycalculating a color difference value for each of the three regions,where the divergence value of a region indicates how different therepresentative color of the region is from the color of the currentlocation. The region with the smallest color difference value may beselected as the region for assignment to which the current location isto be assigned.

After a region has been assigned to the current location, at block 1026,image processing system 204 updates region records for the regionassigned to the current location. Region records maintained for a regionmay identify the first color assigned to the region, the average of thecolors assigned to the region, and/or the area of the region.

At block 1028, if there are more locations to consider, the currentlocation may be incremented to be the next sorted location. The nextsorted location may be identified, at least in part, by accessing thenext location field that corresponds to the current location. Forexample, if the current location is the twelfth location of the inputimage, the next location field corresponding to the twelfth location ofthe input image may identify the next sort location. If all otherlocations of the same color difference value as the current location inthe input image have already been assigned a region, the next locationfield may contain an initialization value, such as 0. In such a case,the next sorted location may be determined by accessing the sort listentry corresponding to the next higher color difference value magnitude.For example, sort list entry 718 corresponds to the color differencevalue magnitude of “2” and contains the value “7.” Thus, the next sortlocation may be the seventh location of the input image. For example, ifthe current location has a color difference value of “1,” the nextlocation may be determined by accessing the sort list entrycorresponding to a color difference value of “2.” If there are no moreentries in sort list that have not been considered, there are no morelocations to consider and the process of FIG. 10 may end.

FIG. 10 illustrates merely one example process for assigning imagelocations to regions. In other embodiments, different steps may beperformed, the ordering of steps may be changed, certain steps may notoccur, or additional steps may occur in addition to the stepsillustrated in FIG. 10.

In an embodiment, after the region assignment process has completed, atleast some of the location IDs contained in the next locations fields oflinked region array 706 field are replaced with different location IDs.Prior to the replacement, the location ID stored in a next locationfield corresponding to a particular image location may identify anotherimage location or linked region array entry corresponding to anotherimage location, where the other image location is associated with thesame color difference value as the particular location. After thereplacement, the location ID stored in a next location fieldcorresponding to a particular image location may identify another imagelocation or linked region array entry corresponding to another imagelocation, where the other image location is assigned to the same regionas the particular location. That is, after the replacement, the nextlocation fields of the linked region array link to other locations ofthe same region instead of other locations associated with the samecolor difference value. Linked region array 1102 is an example linkedregion array which illustrates the values that may be stored in thelinked region array after the modification.

2.4 Alternatives and Extensions

For the purpose of clearly illustrating an example, FIGS. 7, 9, and 11show individual data structures as described above; however, in variousembodiments, each data structure may be implemented as a plurality ofdata structures. Additionally, in other embodiments, each of the datadescribed herein, such as color difference values, next locationinformation, region IDs, and location IDs, may be stored in any type ofdata structure, including, but not limited to an array or a linkedlists.

Alternative embodiments are described throughout the foregoingdescription, and in locations that best facilitate understanding thecontext of the embodiments. Furthermore, the invention has beendescribed with reference to specific embodiments thereof. It will,however, be evident that various modifications and changes may be madethereto without departing from any broader inventive concepts.Therefore, the specification and drawings are, accordingly, to beregarded in an illustrative rather than a restrictive sense.

In addition, in this description certain process steps are set forth ina particular order, and alphabetic and alphanumeric labels may be usedto identify certain steps. Unless specifically stated in thedescription, embodiments are not necessarily limited to any particularorder of carrying out such steps. In particular, the labels are usedmerely for convenient identification of steps, and are not intended tospecify or require a particular order of carrying out such steps.

Functional implementation of the various embodiments described hereinmay be implemented equivalently in hardware, software, firmware, and/orother available functional components or building blocks. No specificlimitation is intended to a particular device or programmatic sequence.Other variations and embodiments are possible in light of aboveteachings.

3. Implementation Mechanism—Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 13 is a block diagram that illustrates a computersystem 1300 upon which an embodiment of the invention may beimplemented. Computer system 1300 includes a bus 1302 or othercommunication mechanism for communicating information, and a hardwareprocessor 1304 coupled with bus 1302 for processing information.Hardware processor 1304 may be, for example, a general purposemicroprocessor.

Computer system 1300 also includes a main memory 1306, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 1302for storing information and instructions to be executed by processor1304. Main memory 1306 also may be used for storing temporary variablesor other intermediate information during execution of instructions to beexecuted by processor 1304. Such instructions, when stored in storagemedia accessible to processor 1304, render computer system 1300 into aspecial-purpose machine that is customized to perform the operationsspecified in the instructions.

Computer system 1300 further includes a read only memory (ROM) 1308 orother static storage device coupled to bus 1302 for storing staticinformation and instructions for processor 1304. A storage device 1310,such as a magnetic disk or optical disk, is provided and coupled to bus1302 for storing information and instructions.

Computer system 1300 may be coupled via bus 1302 to a display 1312, suchas a cathode ray tube (CRT), for displaying information to a computeruser. An input device 1314, including alphanumeric and other keys, iscoupled to bus 1302 for communicating information and command selectionsto processor 1304. Another type of user input device is cursor control1316, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1304 and for controlling cursor movement on display 1312. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1300 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1300 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1300 in response to processor 1304 executing one or moresequences of one or more instructions contained in main memory 1306.Such instructions may be read into main memory 1306 from another storagemedium, such as storage device 1310. Execution of the sequences ofinstructions contained in main memory 1306 causes processor 1304 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any media that storedata and/or instructions that cause a machine to operation in a specificfashion. Such storage media may comprise non-volatile media and/orvolatile media. Non-volatile media includes, for example, optical ormagnetic disks, such as storage device 1310. Volatile media includesdynamic memory, such as main memory 1306. Common forms of storage mediainclude, for example, a floppy disk, a flexible disk, hard disk, solidstate drive, magnetic tape, or any other magnetic data storage medium, aCD-ROM, any other optical data storage medium, any physical medium withpatterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, anyother memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 1302. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1304 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 1300 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 1302. Bus 1302 carries the data tomain memory 1306, from which processor 1304 retrieves and executes theinstructions. The instructions received by main memory 1306 mayoptionally be stored on storage device 1310 either before or afterexecution by processor 1304.

Computer system 1300 also includes a communication interface 1312coupled to bus 1302. Communication interface 1312 provides a two-waydata communication coupling to a network link 1320 that is connected toa local network 1322. For example, communication interface 1312 may bean integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example,communication interface 1312 may be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks may also be implemented. In any such implementation, communicationinterface 1312 sends and receives electrical, electromagnetic or opticalsignals that carry digital data streams representing various types ofinformation.

Network link 1320 typically provides data communication through one ormore networks to other data devices. For example, network link 1320 mayprovide a connection through local network 1322 to a host computer 1324or to data equipment operated by an Internet Service Provider (ISP)1326. ISP 1326 in turn provides data communication services through theworld wide packet data communication network now commonly referred to asthe “Internet” 1328. Local network 1322 and Internet 1328 both useelectrical, electromagnetic or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1320 and through communication interface 1312, which carrythe digital data to and from computer system 1300, are example forms oftransmission media.

Computer system 1300 can send messages and receive data, includingprogram code, through the network(s), network link 1320 andcommunication interface 1312. In the Internet example, a server 630might transmit a requested code for an application program throughInternet 1328, ISP 1326, local network 1322 and communication interface1312.

The received code may be executed by processor 1304 as it is received,and/or stored in storage device 1310, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

4. Example Partitioning Instructions

The following is an example set of instructions for partitioning animage into a plurality of regions. At least a portion of theinstructions may be included in one embodiment of markup regionidentification instructions 210.

-   1. Initialization    -   a. Take the ColorImageInput, BucketSortList, LinkedRegionImage        and ColorDistanceThreshold as inputs.    -   b. Allocate output Objects        -   i. aRegionList=new Region List( )        -   ii. aRegionList.Init( )    -   c. Allocate temporary variables        -   i. Uint32 array pointer LinkedPixelBuffer        -   ii. Uint16 array pointer ColorPixelBuffer        -   iii. Uint32 currentPixelIndex        -   iv. Uint32 array pointer CurrentLinkedPixel        -   v. Uint16array pointer CurrentColorPixel        -   vi. Uint32 currentRegionID.        -   vii. Uint32 nextRegionID.        -   viii. Uint32 aRegionFound        -   ix. Uint32 currentRegionFoundCount.        -   x. Uint64 minDifference        -   xi. Uint64 currentDifference    -   d. Set LinkedPixelBuffer=AddressOf(LinkedRegionImage[0][0])    -   e. Set ColorPixelBuffer=AddressOff(ColorImageInput[0][0]))    -   f. Allocate temporary static arrays        -   i. tempRegionRecord Array        -   ii. Neighborhood Array        -   iii. FoundRegions            -   1. FoundCount                -   a. Number of regions found            -   2. Fields                -   a. Uint32 Region ID            -   3. Size                -   a. Neighborhood.Size-   2. For each non zero value, n, in the BucketSortList // traverse the    bucket sort list    -   a. Set currentPixelIndex=n    -   b. While currentPixelIndex !=0 // traverse the linked list in        LinkedPixelBuffer        -   i. Set CurrentLinkedPixel to            LinkedPixelBuffer+currentPixelIndex        -   ii. Set CurrentColorPixel to            ColorPixelBuffer+(currentPixelIndex*2)        -   iii. Set FoundRegions. FoundCount=0        -   iv. Set aRegionFound=0        -   v. For each element, k, of the NeighboorHood Array            -   1. Set                currentRegionID=CurrentLinkedPixel[NeighboorHood[k]−1]            -   2. If currentRegionID !=0)                -   a.                    currentRegionID=tempRegionRecord[currentRegionID].                    RegionTag                -   b. For each element, j, of the FoundRegions Array                    where (j<FoundRegions .FoundCount)                -   i. If currentRegionID=FoundRegions[k]. RegionID                -   1. Set aRegionFound=currentRegionID                -   ii. If aRegionFound !=0                -   1. Set FoundRegions[FoundRegions .FoundCount].                    Region ID=aRegionFound                -   2. Increment FoundRegions .FoundCount        -   vi. If FoundRegions.FoundCount=0            -   1. Set currentRegionID=tempRegionRecord.RegionCount            -   2. Copy CurrentColorPixel to                tempRegionRecord[currentRegionID].AccumulatedColor            -   3. Set tempRegionRecord[currentRegionID].RegionArea=1            -   4. Set                tempRegionRecord[currentRegionID].CalcAverageDone=FALSE            -   5. Copy CurrentColorPixel to                tempRegionRecord[currentRegionID].FirstFoundColor            -   6. Set                tempRegionRecord[currentRegionID].RegionTag=currentRegionID            -   7. Increment tempRegionRecord.RegionCount            -   8. Set CurrentLinkedPixel[−1]=currentRegionID.        -   vii. if FoundRegions.FoundCount=1            -   1. Set currentRegionID=FoundRegions[0]. RegionID            -   2. Add CurrentColorPixel to                tempRegionRecord[currentRegionID].AccumulatedColor            -   3. Increment                tempRegionRecord[tempRegionRecord.RegionCount].RegionArea.            -   4. Set                tempRegionRecord[currentRegionID].CalcAverageDone=FALSE            -   5. Set CurrentLinkedPixel[−1]=currentRegionID.        -   viii. If FoundRegions.FoundCount=2            -   1. Set currentRegionID=FoundRegions[0]. RegionID            -   2. Set nextRegionID=FoundRegions[1]. Region ID            -   3. If currentRegionID>nextRegionID                -   a. Swap currentRegionID, nextRegionID            -   4. UpdateAverageColor(currentRegionID)            -   5. UpdateAverageColor(nextRegionID)            -   6. If                ColorDifference(tempRegionRecord[currentRegionID].averageColor,                tempRegionRecord[nextRegionID].averageColor)<ColorDistanceThreshold                -   a. tempRegionRecord[nextRegionID].                    RegionTag=currentRegionID.                -   b. Add                    tempRegionRecord[nextRegionID].AccumulatedColor to                    tempRegionRecord[currentRegionID].AccumulatedColor                -   c. Add tempRegionRecord[nextRegionID].RegionArea to                    tempRegionRecord[currentRegionID].RegionArea                -   d. Set                    tempRegionRecord[currentRegionID].CalcAverageDone=FALSE                -   e. Set CurrentLinkedPixel[−1]=currentRegionID.            -   7. Else (set CurrentLinkedPixel region to closest Match)                -   a. If                    (ColorDifference(tempRegionRecord[currentRegionID].averageColor,                    CurrentColorPixel)<ColorDifference(tempRegionRecord[nextRegionID].averageColor,                    CurrentColorPixel)                -   i. Set CurrentLinkedPixel[−1]=currentRegionID.                -   b. Else                -   i. Set CurrentLinkedPixel[−1]=nextRegionID.        -   ix. If FoundRegions.FoundCount>2            -   1. Set minDifference to hex 0x7fffffffffffffff            -   2. Set currentRegionID=0            -   3. For (j=0, j<FoundRegions.FoundCount, j=j+1)                -   a.                    currentDifference=ColorDifference(tempRegionRecord[j].averageColor,                    CurrentColorPixel)                -   b. if (currentDifference<minDifference)                -   i. Set minDifference=currentDifference                -   ii. Set currentRegionID=j            -   4. If (currentRegionID !=0)                -   a. Set CurrentLinkedPixel[−1]=currentRegionID    -   c. Set currentPixelIndex=CurrentLinkedPixel[0]-   3. For each record, n, in tempRegionRecord, where n is Less than    tempRegion Record. RegionCount    -   a. If tempRegionRecord[n].RegionTag=n // this region is the root        of the merged regions        -   i.            tempRegionRecord[n].RegionTag=aRegionList.AddElement(tempRegionRecord[n].FirstFoundColor,            tempRegionRecord[n]. RegionArea, 0) // create an output            region for this one    -   b. Else // this is not a root region        -   i.            tempRegionRecord[n].RegionTag=tempRegionRecord[tempRegionRecord[n].RegionTag].            RegionTag // set tag to the output region index-   4. For Each Pixel, CurrentLinkedPixel, in LinkedPixelBuffer    -   a. Increment currentPixelIndex to the offset for        CurrentLinkedPixel    -   b.        currentRegionID=tempRegionRecord[CurrentLinkedPixel.regionLabelIndex].RegionTag    -   c. CurrentLinkedPixel.regionLabelIndex=currentRegionID // set        tag to the output region index    -   d. CurrentLinkedPixel.nextSortIndex=aRegionList        [currentRegionID].pixelListOffset    -   e. aRegionList        [currentRegionID].pixelListOffset=currentPixelIndex+1-   5. Output the modified LinkedRegionImage and aRegionList,

What is claimed is:
 1. A computer-implemented method, comprising: usingone or more computing devices, receiving an input image comprising adigital image photograph of a customizable product embellished withmarkup; using the one or more computing devices, partitioning the inputimage into a plurality of image regions based on color, wherein thepartitioning comprises: assigning a color difference value to eachlocation of a plurality of locations within the input image and otherlocations neighboring each location, wherein a particular colordifference value is assigned to a particular location; assigning eachlocation of the plurality of locations to an image region of theplurality of image regions in an order based on the assigned colordifference values; selecting an image region for assignment to aparticular location based in part on the color of the particularlocation; using the one or more computing devices, based on thepartitioning, determining information about an item in the input image.2. The computer-implemented method of claim 1, further comprising:determining which regions are assigned to locations adjacent to acertain location of the plurality of locations; assigning the certainlocation to a certain image region of the plurality of image regionsbased, in part, on the determining.
 3. The computer-implemented methodof claim 1, wherein determining which regions are neighboring a certainlocation comprises determining that no regions are associated with thelocations adjacent to the certain location.
 4. The computer-implementedmethod of claim 1, further comprising: determining that a first locationadjacent to a certain location is assigned to a first region and asecond location adjacent to the certain location is assigned to a secondregion different from the first location; assigning the certain locationto a selected region of the first region and the second region based ona determination that a color of the certain location is more similar toa color representing the selected region than a color representing anon-selected region of the first region and the second region.
 5. Thecomputer-implemented method of claim 1, further comprising: determiningthat a certain adjacent location is the only location, of a plurality oflocations adjacent to a certain location, that has been assigned aregion; in response to the determination, assigning the certain locationto a region assigned to the certain adjacent location.
 6. Thecomputer-implemented method of claim 1, further comprising: determiningthat a first location adjacent to a certain location is assigned to afirst region and a second location adjacent to the certain location isassigned to a second region different from the first location;determining whether to merge the first region and the second region,wherein the determining whether to merge the first region and the secondregion comprises comparing a color representing the first region with acolor representing the second region.
 7. The computer-implemented methodof claim 1, further comprising: storing, in a plurality of separate datastructures in computer memory, information describing the plurality ofimage regions, wherein the plurality of data structures include a firstdata structure, which identifies, for each image region of the pluralityof image regions: (a) a representative color value identifying a colorthat represents the image region; (b) a size of the image region; (c) asingle location assigned to the image region.
 8. Thecomputer-implemented method of claim 7, wherein the plurality ofseparate data structures include a second data structure comprising aplurality of entries, wherein each entry of the plurality of entriesrepresents a single corresponding location of the plurality of locationsand identifies either a corresponding next location relative to thecorresponding location, wherein the next location relative to thecorresponding location is a location different than the correspondinglocation and is assigned to the same image region as the correspondinglocation; or a value indicating that there is no next location relativeto the corresponding location.
 9. The computer-implemented method ofclaim 1, further comprising: identifying data representing at least aparticular portion of markup in the input image based on thepartitioning; determining, based on the data, instructions forgenerating a visualization of a customizable product; and sending theinstructions for generating the visualization of the customizableproduct to a web server.
 10. A computer system comprising anon-transitory computer-readable storage medium comprising one or moresequences of instructions which when executed by one or more processorscause the one or more processors to perform: using one or more computingdevices, receiving an input image comprising a digital image photographof a customizable product embellished with markup; using the one or morecomputing devices, partitioning the input image into a plurality ofimage regions based on color, wherein the partitioning comprises:assigning a color difference value to each location of a plurality oflocations within the input image and other locations neighboring eachlocation, wherein a particular color difference value is assigned to aparticular location; assigning each location of the plurality oflocations to an image region of the plurality of image regions in anorder based on the assigned color difference values; selecting an imageregion for assignment to a particular location based in part on thecolor of the particular location; using the one or more computingdevices, based on the partitioning, determining information about anitem in the input image.
 11. The computer system of claim 10, furthercomprising sequences of instructions which when executed by one or moreprocessors cause the one or more processors to perform: determiningwhich regions are assigned to locations adjacent to a certain locationof the plurality of locations; assigning the certain location to acertain image region of the plurality of image regions based, in part,on the determining.
 12. The computer system of claim 10, wherein theinstructions for determining which regions are neighboring a certainlocation comprise sequences of instructions which when executed by oneor more processors cause the one or more processors to determine that noregions are associated with the locations adjacent to the certainlocation.
 13. The computer system of claim 10, further comprisingsequences of instructions which when executed by one or more processorscause the one or more processors to perform: determining that a firstlocation adjacent to a certain location is assigned to a first regionand a second location adjacent to the certain location is assigned to asecond region different from the first location; assigning the certainlocation to a selected region of the first region and the second regionbased on a determination that a color of the certain location is moresimilar to a color representing the selected region than a colorrepresenting a non-selected region of the first region and the secondregion.
 14. The computer system of claim 10, further comprisingsequences of instructions which when executed by one or more processorscause the one or more processors to perform: determining that a certainadjacent location is the only location, of a plurality of locationsadjacent to a certain location, that has been assigned a region; inresponse to the determination, assigning the certain location to aregion assigned to the certain adjacent location.
 15. The computersystem of claim 10, further comprising sequences of instructions whichwhen executed by one or more processors cause the one or more processorsto perform: determining that a first location adjacent to a certainlocation is assigned to a first region and a second location adjacent tothe certain location is assigned to a second region different from thefirst location; determining whether to merge the first region and thesecond region, wherein the determining whether to merge the first regionand the second region comprises comparing a color representing the firstregion with a color representing the second region.
 16. The computersystem of claim 10, further comprising sequences of instructions whichwhen executed by one or more processors cause the one or more processorsto perform: storing, in a plurality of separate data structures incomputer memory, information describing the plurality of image regions,wherein the plurality of data structures include a first data structure,which identifies, for each image region of the plurality of imageregions: (a) a representative color value identifying a color thatrepresents the image region; (b) a size of the image region; (c) asingle location assigned to the image region.
 17. The computer system ofclaim 16, wherein the plurality of separate data structures include asecond data structure comprising a plurality of entries, wherein eachentry of the plurality of entries represents a single correspondinglocation of the plurality of locations and identifies either acorresponding next location relative to the corresponding location,wherein the next location relative to the corresponding location is alocation different than the corresponding location and is assigned tothe same image region as the corresponding location; or a valueindicating that there is no next location relative to the correspondinglocation.
 18. The computer system of claim 16, further comprisingsequences of instructions which when executed by one or more processorscause the one or more processors to perform: identifying datarepresenting at least a particular portion of markup in the input imagebased on the partitioning; determining, based on the data, instructionsfor generating a visualization of a customizable product; and sendingthe instructions for generating the visualization of the customizableproduct to a web server.